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CLAIM + DETAILED DESCRIPTION 



[Claim(s)] 

[Claim 1] The camera operation information storage part which memorizes the camera 
operation information that the photography person operated the camera when photoing video, 
The video recording medium characterized by having at least one of the photographing state 
information storage parts which remember the photographing state information under 
photography obtained by having processed the signal from a sensor to be the Image 
Processing Division information storage part which memorizes the image data acquired by 
having processed the picturized picture. 

[Claim 2] A camera operation information acquisition means to take in the camera operation 
information that the photography person operated the camera when photoing video, An image 
data acquisition means to take in the image data acquired by having processed the picturized 
picture, A photographing state information acquisition means to take in the photographing state 
information under photography obtained by having processed the signal from a sensor, It 
carries out based on at least one of the camera operation information from said camera 
operation information acquisition means, the image data from said image data acquisition 
means, and the photographing state information from said photographing state information 
acquisition means. The still picture extracting apparatus characterized by having a still picture 
extraction means to extract the still picture of at least one sheet out of the video photoed after 
a photography person does photography start operation before carrying out termination 
operation of photography. 

[Claim 3] A camera operation information acquisition means to take in the camera operation 
information that the photography person operated the camera when photoing video, An image 
data acquisition means to take in the image data acquired by having processed the picturized 
picture, A photographing state information acquisition means to take in the photographing state 
information under photography obtained by having processed the signal from a sensor, It 



>://dossierl.ipdl.inpit.go.jp/cgi-b^ 7/18/2008 



JP,07-023322,A(1995) [CLAIM + DETAILED DESCRIPTION] 



Page 2 of 18 



carries out based on at least one of the camera operation information from said camera 
operation information acquisition means, the image data from said image data acquisition 
means, and the photographing state information from said photographing state information 
acquisition means. While recording a still picture extraction means to extract the still picture of 
at least one sheet out of the video photoed after a photography person does photography start 
operation before carrying out termination operation of photography, and the video photoed with 
the imaging device on a video recording medium The video recording device characterized by 
having a still-picture-information record means to record the information on the still picture 
extracted with said still picture extraction means. 

[Claim 4] In order to extract the still picture of at least one sheet out of the video photoed after 
a photography person does photography start operation before carrying out termination 
operation of photography At least one of the photographing state information under 
photography obtained by having processed the signal from image data and a sensor acquired 
by having processed camera operation information, including the zoom of an imaging device, 
photography start operation, etc., and the picturized picture is considered as an input. The still 
picture automatic extraction method characterized by calculating the evaluation value about 
each picture based on still picture extraction knowledge, and extracting a picture with a high 
evaluation value. 

[Claim 5] In order to extract the still picture of at least one sheet out of the video photoed after 
a photography person does photography start operation before carrying out termination 
operation of photography At least one of the photographing state information under 
photography obtained by having processed the signal from image data and a sensor acquired 
by having processed the camera operation information on an imaging device and the 
picturized picture is considered as an input to the picture which satisfies fixed conditions. The 
still picture automatic extraction method with which said camera operation information, said 
image data, and said photographing state information are characterized by extracting the 
picture which satisfied the predetermined extraction condition. 

[Claim 6] In order to extract the still picture of at least one sheet out of the video photoed after 
a photography person does photography start operation before carrying out termination 
operation of photography At least one of the photographing state information under 
photography obtained by having processed the signal from image data and a sensor acquired 
from photography start operation to the picture after definite-period-of-time progress by having 
processed camera operation information, including the zoom of an imaging device etc., and 
the picturized picture is considered as an input. The still picture automatic extraction method 
according to claim 5 with which said camera operation information, said image data, and said 
photographing state information are characterized by extracting the picture which satisfied 
predetermined conditions. 
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[Claim 7] In order to extract the still picture of at least one sheet out of the video photoed after 
a photography person does photography start operation before carrying out termination 
operation of photography As opposed to the picture photoed after the number of pictures which 
multiplied the number of the pictures photoed from photography start operation before the 
termination operation of photography by the rate of a constant ratio At least one of the 
photographing state information under photography obtained by having processed the signal 
from image data and a sensor acquired by having processed camera operation information, 
including the zoom of an imaging device etc., and the picturized picture is considered as an 
input. The still picture automatic extraction method according to claim 5 with which said 
camera operation information, said image data, and said photographing state information are 
characterized by extracting the picture which satisfied predetermined conditions. 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This invention relates to the method of extracting a typical picture 
automatically out of the video photoed with the video camera etc., and its equipment. 
[0002] 

[Description of the Prior Art] As a Prior art, there is a thing of recording the information for 
managing the recorded video on videotape with video, for example with VTR. VISS (VHS 
Index Search System) is explained as an example. With VISS, it is developed in order to 
perform high-speed search in VTR of a VHS method. The control truck which records the VISS 
signal for performing this high-speed search in addition to the video track which records the 
usual picture information exists in videotape. This VISS signal is automatically recorded on a 
control truck, when beginning to record picture information on videotape. Moreover, a VISS 
signal is also recordable to the scene which a user wants to see. Thus, fast forward playback 
called an intro search can be performed using the VISS signal recorded on videotape. If a 
VISS signal is found during fast forward, an intro search will change only a certain time into a 
playback state, and will repeat operation of fast forwarding again after that, to the end of a 
tape. 
[0003] 

[Problem to be solved by the invention] [ however, the VISS signal automatically recorded in 
conventional VTR ] Since it is recorded when beginning to record picture information on 
videotape, when the picture of the portion to which the VISS signal is attached like an intro 
search is extracted, the extracted picture does not fully express the contents of video. 
Moreover, if it is going to give a VISS signal to the portion which fully expresses the contents of 
video, a user has to carry out manually and it will take huge time and effort. 
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[0004] This invention aims at offering the equipment and the video recording medium which 
extract automatically the representation picture which fully expresses the contents of video in 
view of this point. 
[0005] 

[Means for solving problem] The camera operation information storage part which memorizes 
the camera operation information that the photography person operated the camera when the 
video recording medium of this invention photoed video in order to solve the above-mentioned 
technical problem, It has at least one of the photographing state information storage parts 
which remember the photographing state information under photography obtained by having 
processed the signal from a sensor to be the Image Processing Division information storage 
part which memorizes the image data acquired by having processed the picturized picture. 
[0006] Moreover, a camera operation information acquisition means to take in the camera 
operation information that the photography person operated the camera when the still picture 
extracting apparatus of this invention photoed video, An image data acquisition means to take 
in the image data acquired by having processed the picturized picture, A photographing state 
information acquisition means to take in the photographing state information under 
photography obtained by having processed the signal from a sensor, It carries out based on at 
least one of the camera operation information from said camera operation information 
acquisition means, the image data from said image data acquisition means, and the 
photographing state information from said photographing state information acquisition means. 
It has a still picture extraction means to extract the still picture of at least one sheet out of the 
video photoed after a photography person does photography start operation before carrying 
out termination operation of photography. 

[0007] Furthermore, the video recording device of this invention is equipped with a still-picture- 
information record means to record the information on the still picture extracted with said still 
picture extraction means while it records the video photoed with the imaging device on a video 
recording medium. 
[0008] 

[Function] Camera operation information, including the zoom of the photography person at the 
time of photography in the above composition etc., Image data acquired by performing Image 
Processing Division, such as a focus, the reliability of an iris, a position of a photographic 
subject, and an obstacle's existence situation, The picture which is obtained from a sensor and 
for which it evaluates, for example based on photographing state information, including a 
bread etc., and an evaluation value is high, or certain conditions are satisfied is extracted out 
of video. By this, the still picture extracted becomes what fully expressed the contents of video. 

[0009] Moreover, it becomes possible to search a representation picture easily or to output it 
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based on still picture information, by recording the information on the extracted still picture with 

video. 

[0010] 

[Working example] The work example of the video recording medium of this invention is shown 
in drawing i 1 . Although drawing 1 shows videotape as an example of a video recording 
medium, they may be other recording media, such as a videodisk and an IC memory. As 
shown in drawing 1 , corresponding to the video signal, the extraction information on a 
representation picture is recorded on videotape per frame with the video signal. The extraction 
information on a representation picture is camera operation information, image data, and 
photographing state information here. Camera operation is recording start operation, zoom 
operation, etc. at the time of taking a photograph with a video camera, recording start 
operation information is information which sets Flagg and shows the recording starting point, 
when recording start operation is performed, and zoom operation information is information 
showing zoom magnifying power. Both recording start operation information and zoom 
operation information is also information detectable based on button operation of a video 
camera. Image data is automatic or information which man involved and carried out extraction 
processing based on the video signal picturized with the image sensor. For example, the 
frequency and the size of a high frequency component of a video signal for which it asked in 
order to perform focal control, Or they are the state of the thing and backlight which extracted 
information, including the position of a photographic subject field, a size, etc., from the inter- 
frame difference value which searched for the difference between the luminance signal in inter- 
frame, or a chrominance signal, or the video signal, or a fault follow light and also a gamma 
correction value, color temperature, etc. The motion information on cameras, such as panning 
which photographing state information is information which detected the photographing state of 
the camera by the sensor, and was detected, for example with the angular velocity sensor, Or 
the amount of object light by a light volume sensor and the lens which was stopped down and 
detected by the valve travel sensor extract, and there is valve travel or a focal distance by lens 
position detection. 

[001 1] By equipping a video recording medium with the above representation extracted image 
information, it becomes possible to extract the representation picture in video so that 
subsequent work examples may explain. 

[0012] Next, the 1st work example of the still picture extracting apparatus of this invention is 
explained. The block diagram of this example is shown in drawing 2 . This example records the 
information for extracting a representation picture with a video signal on the video recording 
medium. The extraction information on a representation picture is read and evaluated from a 
video recording medium, and a typical picture is extracted as a still picture based on an 
evaluation result out of 1 cut (lump of the video continuously photoed after carrying out 
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recording start operation in a camera before carrying out termination operation of recording). 
drawing 2 - 1 - a regenerative signal input part and 2 - a camera operation information 
acquisition part and 3 -- as for a representation extracted-image-information evaluation part 
and 7, a photographing state information acquisition part and 5 are [ a representation picture 
memory part and 9 ] output units the representation picture Management Department and 8 a 
video signal acquisition part and 6 an image data acquisition part and 4. Operation of each part 
in the above composition is explained in detail below. 

[0013] First, the information recorded on the video recording medium is reproduced and 
inputted into the regenerative signal input part 1. In the camera operation information 
acquisition part 2, the image data acquisition part 3, and the photographing state information 
acquisition part 4, camera operation information, image data, and photographing state 
information are read from the regenerative signal inputted into the regenerative signal input 
part 1, respectively. Each coded information is decoded in this example. As opposed to each 
picture in the same cut until the representation extracted-image-information evaluation part 6 
detects the recording start operation information included in camera operation information and 
it detects the following recording start operation information It is evaluated whether it is a 
picture with each picture suitable as a representation picture of a cut based on information, 
image data, and photographing state information other than the recording start operation 
information in camera operation information. About the picture estimated to be suitable as a 
representation picture, the representation picture Management Department 7 captures the 
image of one frame from the video signal acquisition part 5, and memorizes in the 
representation picture memory part 8. The output unit 9 takes out and outputs the 
representation picture memorized by the representation picture memory part 8, and are a 
display, a printer, etc. In addition, the work example of the below-mentioned still picture 
automatic extraction method explains operation of the representation extracted-image- 
information evaluation part 6 in detail. 

[0014] The information for extracting a representation picture beforehand is recorded on the 
video recording medium with the video signal, and the above work example explained the case 
where read the extraction information on a representation picture from a video recording 
medium, and a representation picture was extracted. However, even when a part or all of the 
information for extracting a representation picture does not exist in a video recording medium, 
by processing the video signal recorded on the video recording medium, the information for 
extracting a representation picture can be acquired and a representation picture can be 
extracted based on the acquired information. The still picture extracting apparatus of the 
following work example [ 2nd ] explains this in detail. 

[0015] The 2nd work example acquires all the representation extracted image information only 
from a video signal. The example of composition of the equipment which acquires 
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representation extracted image information from a video signal to drawing 3 is shown. 10 an 
inter-frame difference value detecting element and 1 1 a memory and 12 by drawing 3 The 
amount detecting element of change, 13 a cut change detecting element and 14 a camera 
work detecting element and 15 A motion vector detecting element, 16 -- as for the 
characteristic quantity extraction part in a field, and 20, a photographic subject information 
detecting element and 18 are [ a high pass filter and 22 ] average calculation parts a focal 
information detecting element and 21 a motion area detecting element and 19 a camera work 
parameter presumption part and 17. Operation of each part in the above composition is 
explained in detail below. 

[0016] First, operation of the inter-frame difference value detecting element 10 and the cut 
change detecting element 13 is explained. The inter-frame difference value detecting element 
10 consists of an amount detecting element 12 of change which asks for the difference of a 
video signal by inter-frame [ which follows the memory 1 1 for delaying one video signal ]. The 
signal which searches for the inter-frame difference which video follows performs the 
difference operation of the inter-frame picture signal which continues by a pixel unit in the 
amount detecting element 12 of change using a brightness value, a rgb value, etc., and 
outputs it as an inter-frame difference value in quest of total of the difference value for every 
pixel. The cut change detecting element 13 carries out threshold processing to the inter-frame 
difference value calculated by the inter-frame difference value detecting element 10. That is, 
the comparison with a predetermined threshold and an inter-frame difference value is 
performed, and when an inter-frame difference value is larger than a threshold, an image 
content thinks that it is changing a lot by inter-frame [ of two sheets ], and it is judged that there 
was a cut change in the portion. In a video camera, since a cut change arises by performing 
recording start operation, recording start operation can be presumed by detecting a cut change 
from a picture signal conversely. Therefore, in the cut change detecting element 13, when the 
inter-frame difference value exceeding a threshold is detected, recording start operation 
information is outputted. In addition, the composition of the inter-frame difference value 
detecting element 10 shown by drawing 3 may be an example, and other composition as 
shown by drawing 4 may be used for it. The color histogram detecting element which asks for 
a color histogram [ in / in 44 / one frame of a video signal ] by drawing 4 , the histogram 
memory which memorizes the histogram which calculated 45, and 46 are histogram difference 
detection parts which detect the difference in a color histogram by inter-frame [ continuous ]. 
Although every pixel is not measured by inter-frame but the whole frame compares with the 
composition shown in drawing 4 , it is good also as composition which divides a screen into 
two or more blocks, and asks for inter-frame difference in a block unit. 
[0017] Next, the camera work detecting element 14 is explained. First, operation of the motion 
vector detecting element 15 is explained. Drawing 5 is a figure for explaining the position in the 
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screen of the motion vector to detect. Drawing 5 (a) is the figure which arranged the straight 
line of M and N book in the shape of a grid to level and a perpendicular direction by the full 
screen, and M.N intersections show the position of the motion vector which should be 
detected. It calls a lattice point M.N intersections below and is level and perpendicular, and is a 
lattice point (i, j) about i and the j-th lattice point respectively. (1 <=i<=M, 1 <=j<=N) 
It calls. 

[0018] The motion vector in a lattice point position chooses two or more representative points 
by this example around each lattice point, and it asks by representative point matching. 
Drawing 5 (b) is the figure which expanded the neighborhood of a lattice point (i, j) of drawing 5 
(a), and shows the physical relationship of a lattice point and the representative point of - (2- 
m+1) (2-n+1) ** of the circumference of it. It is a representative point (i, j, k, I) about k and the I- 
th thing, respectively to the level among the representative points of the following and a lattice 
point (i, j), and a perpendicular direction. (-m<=k<=m, -n<=l<=n) 

It calls. A representative point (i, j, 0, 0) is equal to a lattice point (i, j) so that drawing 5 (b) may 
show. 

[0019] How to ask below for a motion vector using drawing 6 which showed the concrete block 
diagram of the motion vector detecting element 15 is explained. 

[0020] The input of the motion vector detecting element 15 is a video signal, and assumes that 
it is set as the r frame so that it may be inputted once (r: predetermined number). Here, the 
picture of a certain time t is made into the picture of the 0th frame, and the picture of time 
(t+tau) will be henceforth called the picture of a ** (30andtau) frame eye. 
[0021] The picture of the Rth frame should be inputted now. In BPF23, a band pass filter lets 
an input picture pass first. The value of the picture after BPF processing in a coordinates 
position (x, y) is set to I (x, y) here. 

[0022] On the other hand, as for the representative point value memory part 24, the value of 
the representative point of the picture in front of the r frame (i.e., the BPF processing back of a 
** (R-r) frame eye) is memorized. Namely, value Y(i, j, k, I) =l of a representative point (i, j, k, I) 
([ pos_x (i, k) and ]) pos_y(j, I) 1 <=i<=M, 1 <=j<=N, -m<=k<=m, x-coordinate pos_y of a - 
n<=l<=npos_x(i, k):representative point (i, j, k, I) (i, k): y coordinates of a representative point (i, 
j, k, I) are memorized. 

[0023] The matching part 25 inputs the value Y of the representative point in front of [ the 
representative point value memory part 24 to ] the r frame (i, j, k, I) for the picture I after BPF 
processing (x, y) from BPF23, and asks for the motion vector in each lattice point by 
representative point matching. About a lattice point (i, j), namely, [0024] 
[Mathematical formula 1] 

£ f {Y(i,j,k,l)-I (pos_x(i,0)+g, pos_y(j,0)+h) ) 
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[0025] A motion vector (g, h) can be found by searching for g and h used as ****** within the 
limits of - (2andG) (2andH) (-G<=g<=G, -H<=h<=H). 

[0026] The contents of the representative point value memory part 24 are updated after 
processing of the matching part 25 is completed. Specifically, the value in the representative 
point of the picture after the Rth-frame BPF processing is recorded using coordinates pos_x (i, 
j, k, I) of the representative point memorized in the representative point position memory part 
26, pos_y(i, j, k, I) 1 <=i<=M, 1 <=j<=N, -m<=k<=m, and -n<=l<=n. 

[0027] It can ask for a motion vector from the picture, the picture inputted as mentioned above 
and the picture in front of the r frame, of two sheets. 

[0028] Next, in the camera work parameter presumption part 16, how to presume a camera 
work parameter from a motion vector is explained. 

[0029] The camera work which can be presumed from video can consider the level of a 
camera, a vertical change (panning, tilting), change (zooming) of a camera field angle, change 
(tracking, booming, Dolly Inge) of the position of horizontal, vertical, and [ of a camera ] order, 
etc. By this example, since it is easy, panning, a tilting, and the method of presuming three 
kinds of operations of zooming are explained. 

[0030] First, it is considered how the point projected on the imaging surface of the camera 
moves by the three above-mentioned kinds of camera works. Drawing 7 is the figure showing 
the imaging surface of a camera, and the physical relationship of a photographic subject, 
expresses the three-dimensions coordinates of the space of a camera with (x, y, z), and 
expresses the two-dimensional coordinates on an imaging surface with (X, Y). Moreover, make 
the position of a camera into the starting point of three-dimensions coordinates, and let the 
optical axis of a camera be a z axis. An imaging surface is located in z=F (F: focal length), and 
it is shown that coordinates u1= (x1, y1, z1) of the arbitrary points of a photographic subject is 
projected on U1= (X1, Y1) of an imaging surface. The relation between the coordinates of a 
photographic subject, and the coordinates on an imaging surface here [0031] 
[Mathematical formula 2] 
Xl=F-xl/zl 
Yl = F*yl/zl 

[0032] It can come out and express. The migration on the imaging surface of the coordinates 
of a photographic subject by zooming is first considered using the coordinates of drawing 7 
Drawing 8 (a) shows zooming which happens by change of a focal length. As shown in this 
figure, when a focal length changes from F to F', projection of the photographic subject of u1 
moves to U2= (X2, Y2) from U1= (X1, Y1). 

[0033] However, (several 2) to U2 fills U2=U1, and F7F=f-U1, however f=F7F. 

[0034] The case of panning and a tilting is considered using drawing 8 (b) the same way. 

Panning and a tilting are equal to operation of rotating a camera about a y-axis and an x-axis, 
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respectively. As shown in this figure, when a camera rotates only thetax about an x-axis, the 
coordinates u1 in the space of a photographic subject move to u3. However, u3 fills (several 
3). 

[0035] 

[Mathematical formula 3] 



u3=ul 



1 0 0 
0 cos 9 x —sin 9 x 
0 sin 9 x cos 9 x 



[0036] If rotation angle thetax about x assumes that it is small enough, the relation of X3=X1 
and Y3=Y1+ F-theta x will be drawn from the relation (several 2) to coordinates U3= on the 
imaging surface after migration (X3, Y3). When this is generalized, in the camera operation 
rotated [ as opposed to / both / an x-axis and a y-axis ], the relations before and behind 
operation of arbitrary coordinates are U3=U1+P, however P= (px, py). 
px, py: It can express the rotation component of an x-axis and a y-axis. 
[0037] The above thing shows that coordinates 111= before and behind camera operation (X1, 
Y1) and U'= (X', Y) fill U-f-U+P to the general camera operation which compounded zooming, 
panning, and a tilting. Henceforth, f is called a zoom element and P is called a rotation vector. 
[0038] Therefore, by asking for a zoom element and a rotation vector shows that the control 
input of a camera can be presumed. 

[0039] How to presume a zoom element and a rotation vector is explained from the motion 
vector for which below was asked by the motion vector detecting element 15. Here, the motion 
vector asked for the position (two-dimensional coordinates) by Ui, j, and the motion vector 
detecting element 15 is set to vi and j about a lattice point (i, j). 

[0040] When camera operation of the zoom element f and the rotation vector P takes place 
now, lattice points (i, j) are U'i, j(f, P) =f-Ui, and j. It should move to the position of +P. 
Therefore, what is necessary is just to ask for f and P from which the actually moved position 
Ureali, j=Ui, j+vi, and error E(f, P) =sigma(U'i, j(f, P)-Ureali, j)2 with j become the minimum, in 
order to presume f of the camera operation which actually took place, and P. Since Error E is a 
secondary type about f and P, f and P which are made into the minimum uniquely Error E 
[0041] 

[Mathematical formula 4] 

f = £i>i<U rea 'U. Uu> <£i, jU real i,j, Li. jUi, j>/M/N 
£i,j<Ui,j, Ui,j>-<£i, jUi.j, Li, jUi, j>/M/N 

P= {£i,jU r "'i,j-f -Li, jUi.j) /M/N 
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[0042] It is decided. However, <-, -> show an inner product. Therefore, in the camera work 
parameter presumption part 16, zooming, panning, and each camera work parameter of a 
tilting can be presumed by inputting the motion vector vi, j, and the lattice point position Ui and 
j from the motion vector detecting element 15, and calculating f and P by (several 4). 
[0043] Next, operation of the photographic subject information detecting element 17 is 
explained. The photographic subject information detecting element 17 extracts photographic 
subject information, including the position of a photographic subject, a size, a color, etc., in the 
state where tracking of the photographic subject is carried out, with a video camera. That is, 
the characteristic quantity in a field is extracted from a motion area in the characteristic 
quantity extraction part 19 in a field to the case where panning was detected by the camera 
work detecting element 14, and a motion area is able to be further detected by the motion area 
detecting element 18. Operation in the motion area detecting element 18 is explained further in 
full detail. 

[0044] Motion vector **** of the camera by the motion vector vi of M.N lattice points in the 
screen detected by the motion vector detecting element 15 in the motion area detecting 
element 18, j, and panning detected by the camera work detecting element 14 It is inputted. 
Extract the lattice point which fills (several 5) with the motion area detecting element 18, and 
based on the connection relation of the extracted lattice point Panning [0045] 
[Mathematical formula 5] 

I vi.j-Vpt >e tztzU eiiBf&Offi 

[0046] A different field from the motion vector of the camera boiled and twisted is extracted. In 
the characteristic quantity extraction part 19 in a field, a center-of-gravity position, area, and a 
color are extracted from the motion area detected by the motion area detecting element 18 as 
characteristic quantity in a field. 

[0047] Next, operation of the focal information detecting element 20 is explained. The focal 
information detecting element 20 is for detecting the pin dotage state of a picture, and is 
carried out based on the quantity of the high frequency component of a picture. That is, after 
the picture has faded by the focal gap of a lens etc., the value of the high frequency 
component of a video signal becomes small. For this reason, it has composition which takes 
out the high frequency component of a picture with the high pass filter 21 , and calculates the 
average of the high frequency component in the whole screen or the appointed field in the 
average calculation part 22. 

[0048] The information for extracting a representation picture can be acquired by processing a 
video signal as mentioned above. Although this example indicated neither a gamma correction 
value, color temperature, a backlight nor a fault follow light state, the amount of object light, 
etc., these information can also be acquired by processing a video signal. After acquiring the 
information for extracting a representation picture, about the composition and the technique of 
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extracting a representation picture based on the acquired information, it is the same as that of 
the 1st work example of a still picture extracting apparatus, and explanation is omitted. 
[0049] [ equipment / the equipment which acquires the representation extracted image 
information explained above acquired representation extracted image information based on the 
video signal read from the video recording medium, when the information for extracting a 
representation picture did not exist in a video recording medium, but ] Representation 
extracted image information can also be similarly acquired with a video camera based on the 
video signal taken in from the image sensor during photography. The composition in this case 
is the same as that of drawing 3 , and although explanation is omitted, when representation 
extracted image information detectable by the sensor with which the video camera was 
equipped exists, it cannot be overemphasized that it is not necessary to acquire representation 
extracted image information from a video signal. Furthermore, you may record the 
representation extracted image information acquired during photography with the video 
camera on a video recording medium with a video signal. 

[0050] Next, the composition of the work example of the video recording device of this 
invention is shown in drawing 9 . This example is the composition which equipped the video 
camera with the still picture extracting apparatus, determines the still picture extracted as a 
representation picture during photography with a video camera, and records the information on 
the still picture extracted while recording a video signal on a video recording medium. The 
video recording device 27 of this invention consists of the camera operation information 
acquisition part 28, the image data acquisition part 29, the photographing state information 
acquisition part 30, the video signal acquisition part 31, the representation extracted-image- 
information evaluation part 32, the still-picture-information Records Department 33, and the 
video signal Records Department 34 by drawing 9 . Operation of each part in the above 
composition is explained in detail below. 

[0051] The camera operation information acquisition part 28 is a portion which acquires 
information, including the recording start operation at the time of taking a photograph with a 
video camera, zoom operation, etc. Recording start operation information is information which 
sets Flagg and shows the recording starting point, when recording start operation is performed, 
and zoom operation information is information showing zoom magnifying power. Both 
recording start operation information and zoom operation information is also detected based 
on button operation of a video camera. The image data acquisition part 29 is the portion which 
acquires the information which processed the video signal picturized with the image sensor. 
For example, the frequency and the size of a high frequency component of a video signal for 
which it asked in order to perform focal control, Or the state of information, including the 
position of the inter-frame difference value which searched for the difference between the 
luminance signal in inter-frame or a chrominance signal, or the photographic subject field for 
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which it asked from the video signal, a size, etc., a backlight, or a fault follow light and also a 
gamma correction value, color temperature, etc. are extracted. The photographing state 
information acquisition part 30 is the portion which acquires the information which detected the 
photographing state of the camera by the sensor. For example, the amount of object light by 
the motion information on cameras, such as panning detected with the angular velocity sensor, 
or a light volume sensor and the lens which was stopped down and detected by the valve 
travel sensor extract, and valve travel or a focal distance by lens position detection is acquired. 

[0052] As opposed to each picture in the same cut until the representation extracted-image- 
information evaluation part 32 detects the recording start operation information included in 
camera operation information and it detects the following recording start operation information 
It is evaluated whether it is a picture with each picture suitable as a representation picture of a 
cut based on information, image data, and photographing state information other than the 
recording start operation information in camera operation information. Still picture information 
is recorded on the still-picture-information recording medium of the recording medium 35 
through the still-picture-information Records Department 33 about the picture estimated to be 
suitable as a representation picture. In addition, the work example of the below-mentioned still 
picture automatic extraction method explains operation of the representation extracted-image- 
information evaluation part 32 in detail. 

[0053] Below, the still picture information recorded at the still-picture-information Records 
Department 33 is explained further. The still picture itself estimated to be suitable as a 
representation picture by the representation extracted-image-information evaluation part 32 
with still picture information Or he is Flagg who gave the video corresponding to the storing 
position information on video recording media corresponding to the thing which carried out 
Image Processing Division, or a still picture, such as reduction, or a still picture. 
[0054] When still picture information is the still picture itself or the reduced picture The position 
where the record positions on the video recording medium which records the video photoed 
with the camera by the video signal acquisition part 31 and the video signal Records 
Department 34 differ, or the video recording medium which records video records the picture 
which is still picture information on a different recording medium. For example, when a video 
recording medium is videotape, only still picture information is recorded on the IC memory 
which summarized only the picture of still picture information, and recorded on the head 
portion of a tape, or the termination portion of the tape, or it had apart from the tape. 
[0055] [ in the case of the storing position information on the video recording medium 
corresponding to a still picture in still picture information ] A different position from the record 
position on the video recording medium which records the video photoed with the camera by 
the video signal acquisition part 31 and the video signal Records Department 34, Or the video 
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recording medium which records video records the storing position information on the video 
recording medium corresponding to the still picture which is still picture information on a 
different recording medium. 

[0056] In the case of Flagg whom still picture information gave to the video corresponding to a 
still picture, still picture information is recorded on the same position as the record position on 
the video recording medium which records the video photoed with the camera by the video 
signal acquisition part 31 and the video signal Records Department 34. That is, Flagg of still 
picture information is recorded on the head portion of a video signal recorded, for example per 
one frame. 

[0057] While recording the video signal photoed with the video camera as mentioned above on 
a video recording medium, a representation picture is extracted out of the photoed video, and 
the still picture information of a representation picture is recorded on a recording medium. It 
becomes possible to read the still picture information recorded by this and to output a 
representation picture to a display or a printer. 

[0058] Next, the work example of the still picture automatic extraction method in this invention 
is explained. A still picture automatic extraction method is the processing method in the 
representation extracted-image-information evaluation part 6 of drawing 2 , and the 
representation extracted-image-information evaluation part 32 of drawing 9 . 
[0059] The still picture automatic extraction method of this invention extracts a typical picture 
automatically as a still picture out of the video continuously photoed after carrying out 
recording start operation before carrying out termination operation of recording. A typical 
picture means the picture evaluated and selected based on the intention of a photography 
person, the state of the photoed picture, and the state of a photographic subject here. 
[0060] An intention of a photography person is reflected in camera works, such as zoom and a 
bread. That is, while zooming in, it is the case where the photographic subject currently 
observed exists in a screen, and is considered an important picture. Moreover, while carrying 
out the bread, it is in the midst of moving to another scene from a certain scene, and it is 
thought that it is not important. Furthermore, when the photographic subject which is carrying 
out tracking even when the bread is being carried out exists, it is thought that it is important. 
Thus, it is desirable to presume an intention of a photography person from camera work, and 
to extract an important portion as a representation picture. 

[0061] The state of the photoed picture means states the state of a fault follow light when the 
image state which faded and iris control in case focal control is not well performed at the time 
of photography are unsuitable, or a backlight, when gamma correction is still more unsuitable, 
an iris diaphragm, the state under adjustment of a focus, etc. These image states can be 
judged based on the information on the focal control at the time of photography with a video 
camera, or iris control, or a gamma correction value. Moreover, even if it is the case where 
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there is no information on focal control, iris control, and gamma correction, it is possible to ask 
by processing a video signal. It is desirable to evaluate these image states and for an image 
state to extract a good thing as a representation picture. 

[0062] The state of a photographic subject means states when a flash shines during an 
obstacles' when people's cross camera front's during state [, such as a position of the 
photographic subject currently photoed and a size, ] and photography existence situation, and 
photography, the state where the spotlight is irradiated by the photographic subject, etc. It is 
more desirable for the one where area is larger to have a desirable position in the center of a 
camera, and for an obstacle not to exist about the position or size of a photographic subject. 
Moreover, it is more desirable not to extract a picture when a flash shines as a representation 
picture. Moreover, the picture by which the spotlight is irradiated is an attention picture and 
extracting as a representation picture is desirable. Here, in the 3rd work example, the 
photographic subject information detecting element 17 of drawing 3 explains the detection 
method of the position of a photographic subject, or area. Moreover, about the detection 
method of a flash or an obstacle, it is detectable based on the inter-frame difference value 
calculated by the inter-frame difference value detecting element 10 of drawing 3 . That is, since 
an inter-frame difference value changes suddenly, a flash is detectable based on a 
predetermined threshold. Since in the case of an obstacle an inter-frame difference value 
changes when an obstacle comes out from the time of an obstacle entering into a screen, and 
a screen, After an inter-frame difference value exceeds a predetermined threshold, when an 
inter-frame difference value exceeds a threshold again within predetermined time, it can detect 
under the condition that an obstacle exists in a screen. Moreover, irradiation of a spotlight is 
detectable based on the amount of object light. 

[0063] Below based on the knowledge for extracting the above representation pictures, the 
extraction technique of a concrete representation picture is explained. The example of 
composition of this example is shown in drawing 10 As for a weighting adder unit and 37 a 
gate part and 39 are maximum detecting elements a gate signal generating part and 38 36 in 
drawing 10 . Zoom magnifying power and photographic subject information are inputted into 
the weighting adder unit 36, and dignity is attached and added to each signal. Here, 
photographic subject information is information acquired while carrying out tracking of the 
photographic subject with the camera, and is carried out based on the position and size of a 
photographic subject at the time of tracking. It is made for photographic subject information to 
become such a big value that the area of a photographic subject is so large that an object 
position is close to the center of a camera. The gate part 38 performs ON of a switch, and OFF 
based on the gate signal of the gate signal generating part 37. The maximum detecting 
element 39 detects the maximum of a value inputted from the gate part 38. 
[0064] The gate signal generating part 37 generates a gate signal based on a high frequency 
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component value, an inter-frame difference value, etc. of a pan-signal and a video signal. The 
generating method of a gate signal is shown in drawing 1 1 . Drawing 1 1 (a) is a pan-signal, 
under panning is 0, and the time of not having carried out a bread is a signal used as 1. (b) is 
the high frequency component value of a video signal, and it means being in the state in which 
the picture faded, so that a value is small, (c) carries out threshold processing, binary-izes the 
signal of (b), and, in below a threshold, makes it 0. . (d) is an inter-frame difference value, (e) 
carries out threshold processing of the signal of (d), in more than a threshold, it is made into 0, 
and after the signal of (d) exceeds a threshold further, when a threshold is again exceeded in 
predetermined time, it performs processing which also sets the section between zero to 0. That 
is, when an inter-frame difference value becomes large independently, it is judged as what the 
abnormalities in a picture with a flash etc. generated, and an inter-frame difference value sets 
to 0 only the period which is more than a threshold. However, as mentioned above, when an 
obstacle passes through a camera front, in order that an inter-frame difference value may take 
two or more peaks, the period when an obstacle exists in a screen is set to 0 even if an inter- 
frame difference value is below a threshold. A gate signal is generated by taking AND of three 
signals of (a), (c), and (e) binary-ized as mentioned above. 

[0065] By asking for the picture from which an evaluation value turns into maximum with the 
composition shown by drawingjo as mentioned above While removing the period which is 
carrying out the bread, the period when the picture faded, and the period when a flash and an 
obstacle exist further, zoom magnifying power is high and the picture by which the 
photographic subject is greatly reflected to middle of the screen can be extracted from from as 
a representation picture. In addition, the whole 1 cut may detect maximum by the maximum 
detecting element 39, or they may be two or more sections in 1 cut. 
[0066] In addition, although one evaluation value is calculated from two or more inputs in 
drawing 10 by composition of the weighting adder unit 36. the gate signal generating part 37 
and the gate part 38 The thing based on rules, such as fuzzy reasoning instead of what was 
restricted to this composition, and the composition for which it asks by a neural network are 
also possible. Furthermore, although a gamma correction value, the amount of object light, a 
backlight or a fault follow light state, and processing of the information extract and concerning 
valve travel and a focal distance were not shown in the composition of this example, these 
signals can be used similarly. That is, you may generate a gate signal a gamma correction 
value, while extracting and changing the value of valve travel or a focal distance, and so that it 
may not extract as a representation picture in a backlight or a fault follow light state. Moreover, 
it detects that the spotlight is irradiated from the amount of object light, and may be made to 
make an evaluation value high. 

[0067] [ composition ] although the composition of the work example of the still picture 
automatic extraction method explained above evaluates to all the video photoed after a 
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photography person does photography start operation before carrying out termination 
operation of photography and is extracting the representation picture The composition of 
extracting the picture at the time of performing evaluation from the picture after predetermined 
time progress after the photography person did photography start operation, and satisfying 
predetermined conditions as a representation picture may be used. Below, the work example 
of this composition is explained. 

[0068] The composition of this example is shown in drawing 12 . As for 40, a gate part and 42 
are evaluation parts a timer and 41 in drawing 12 . A timer 40 measures the elapsed time after 
photography start operation is performed, and when the definite period of time has passed 
since the photography start, it generates a gate signal so that the gate of the gate part 41 may 
be opened. The evaluation part 42 evaluates whether the high frequency component value and 
inter-frame difference value of the focus which passed the gate part 41 have satisfied 
conditions. The conditions in the evaluation part 42 are conditions that the high frequency 
component value of a focus is more than a predetermined threshold, and an inter-frame 
difference value is below a predetermined threshold. When satisfied with the evaluation part 42 
of conditions, evaluation after it is stopped, and the still picture at the time of conditions being 
satisfied is extracted as a representation picture. In addition, although the signal used for 
evaluation is only 2, the high frequency component value of a focus, and an inter-frame 
difference value, in this example, you may use other signals, such as a bread and zoom. 
[0069] Furthermore, the composition of another work example of a still picture automatic 
extraction method is explained. The composition of this example is shown in drawing 13 , the 
same number is attached to the same thing as drawing 12 , and explanation is omitted. This 
example performs evaluation from the number of pictures which multiplied the number of the 
pictures photoed from photography start operation before the termination operation of 
photography by the rate of a constant ratio, and extracts the picture at the time of satisfying 
predetermined conditions as a representation picture. The composition of dr awing 13 
estimates the picture after an intermediate frame to the picture photoed from photography start 
operation before the termination operation of photography. For this reason, in the intermediate 
frame detecting element 43, when the intermediate frame of the frame photoed by the 
termination operation of photography is detected from photography start operation and an 
intermediate frame is detected, a gate signal is generated so that the gate of the gate part 41 
may be opened. Operation of the gate part 41 and the evaluation part 42 is the same as that of 
the composition of drawing 12 , and explanation is omitted. 
[0070] 

[Effect of the Invention] As explained above, the video recording medium of this invention 
becomes possible [ extracting easily the representation picture in the video recorded on the 
video recording medium based on extracted image information ] by having extracted image 
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information. 

[0071] Moreover, the still picture extracting apparatus of this invention becomes possible 
[ extracting the still picture which fully expresses the contents of video out of video based on 
extracted image information as a representation picture ], and can grasp the contents of video 
in a short time. 

[0072] Furthermore, [ a recording device ] while the video recording device of this invention 
records the video signal photoed with the camera on a video recording medium It becomes 
possible to read the recorded still picture information and to output a representation picture to 
a display or a printer at high speed by extracting a representation picture out of the photoed 
video, and recording the still picture information of a representation picture on a recording 
medium. 



[Translation done.] 
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